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TITLE OF THE INVENTION 

Zero excess bandwidth modulation 
CROSS REFERENCE TO RELATED PATENTS/PATENT APPLICATIONS 

The present U.S. Utility Patent Application claims priority pursuant to 35 
5 U.S.C. § 119(e) to the following U.S. Provisional Patent Application which is hereby 
incorporated herein by reference in its entirety and made part of the present U.S. 
Utility Patent Application for all purposes: 

1. U.S. Provisional Application Serial No. 60/453,051, entitled "Zero 
excess bandwidth modulation," (Attorney Docket No. BP 2431), filed March 8, 2003 
10 (03/08/2003), pending. 

BACKGROUND OF THE INVENTION 
TECHNICAL FIELD OF THE INVENTION 
The invention relates generally to communication systems; and, more 
particularly, it relates to modulation of signals that are transmitted within such 
15 communication systems. 

DESCRIPTION OF RELATED ART 
Data communication systems have been under continual development for many 
years. A design imperative, particularly in the more recent days of broadband 
communications, is to maximize the amount of data (or information) that can reliably 
20 be communicated within a given period of time over a communication channel 
between a first point and a second point, e.g., the locations of a communication 
transmitter and a communication receiver. 

It is well known since Shannon's original work in the late 1940's and 
subsequent interpretations that the maximal data rate supported by a communication 
25 channel increases linearly with the channel bandwidth and logarithmically with the 
SNR (signal-to-noise ratio) of the received signals. More precisely, the supported data 
rate depends linearly on the modulation rate, which is limited by the available channel 
bandwidth. Further, the supported data rate depends logarithmically on the SNR 
reduced by the so-called "SNR gap to capacity," which depends on the sophistication 
30 of information encoding into signal waveforms at communication transmitters and 
decoding approaches employed in communication receivers. 
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A very significant portion of communication research has been devoted to 
reducing the SNR gap to capacity. For uncoded modulation, the gap to capacity is 
about 9 dB at moderately low error rate. Hence, due to the lack of sophisticated 
coding, for reliable information transmission the SNR must be 9 dB (decibels) higher 
5 than needed in an ideal system with capacity achieving modulation and coding in the 
sense of Shannon's theory. During the past two decades, this gap has been reduced to 
about 3 dB to 5 dB by the introduction of coded modulation, e.g., trellis coded 
modulation, and the extensive use of FEC (forward error correction) coding, e.g., Reed 
Solomon coding. More recently, "capacity approaching" coding techniques like turbo 
10 coding and decoding provide the possibility of reducing the SNR gap to capacity to 
less than 1.5 dB. 

In comparison to work in coding research, efforts to achieve the highest 
possible modulation rate within given channel bandwidth have been of a more 
technical nature. As the modulation rate is increased towards the bandwidth of the 
15 communication channel, undesired ISI (inter-symbol interference) among serially 
transmitted modulation symbols is becoming harder to avoid. Similarly, suppression 
of interference from adjacent channels (ACI = adjacent channel interference) becomes 
more difficult. 

Many of the presently deployed communication systems are based on the 
20 concept of zero-ISI transmission. In these systems, the modulation rate is 
1/T = W/(l + ot), where W is the channel bandwidth and a > 0 is the fraction of 
"excess bandwidth" sacrificed to allow for smooth roll-offs of filter transfer functions 
towards zero outside of the transmission band. The quantity a is therefore also 
referred to as "spectral roll-off factor." Typical values of a are in the range of 0.1 to 
25 0.3. For given target values of ISI and ACI suppression, the complexity of transmit 
and receive filter increases with decreasing value of a. A chosen value of a reflects a 
compromise between filter complexity and efficient use of the available bandwidth. 

FIG. 1 illustrates a prior art communication system operating at a modulation 
rate of 1/T = W /(1 + a), across a communication channel of bandwidth W. In order to 
30 achieve zero ISI, the spectrum of the received signal must satisfy a condition known as 
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the Nyquist criterion. Let H(f ) be the Fourier transform of the received waveform in 
response to sending a single modulation symbol at time zero. This symbol response is 
considered at the output of the overall channel comprising transmit filter(s), 
communication channel, and receive filter(s). With constant delays ignored, the 
5 Nyquist criterion requires that H(f ) and shifted versions of H(f ) add up to a constant 



the symmetric fashion indicated in FIG. 1 around the edges of the 1/T band, which is 
called the Nyquist band. If the factor a were reduced to zero, such that 1/T = W , the 
spectral symbol response H(f ) would have to assume rectangular shape. The required 

10 "brick wall" filtering cannot be realized with finite filter complexity. 

In communication systems that operate using multiple bands, suppressing 
interference from signals transmitted in adjacent bands, in addition to suppressing out- 
of-band noise, is of major concern. FIG. 2 depicts the partitioning of a total bandwidth 
nW into n subbands of bandwidth W. Each subband is used for zero-ISI transmission 

15 at modulation rate 1 / T . 



system with SQRC (square root raised-cosine) transmit and receive filters. An ideal 
communication channel with flat transfer function G c (f ) = 1 and constant PSD (power 

spectral density) N(f) = N 0 (AWGN = additive white Gaussian noise) within the 
20 transmission band of the channel is assumed. The transfer functions of the transmit 



filter, H T (f), and the receive filter, G R (f), are given by H T (f) = G R (f) = V H (f ) , where 




must roll off in 



FIG. 3 illustrates a prior art, complex baseband model of a zero-ISI baseline 



H(f ) is the spectral RC (raised cosine) symbol response 
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of the overall channel. One can easily verify that H(f ) satisfies the Nyquist criterion, 

and J|H T (f fdf = J|G R (f f df = jH(f) df = 1 . 

In this system, complex- valued modulation symbols a n are transmitted at a 
modulation rate of 1/T = W/(l + ot). The modulation symbols are independently and 
5 identically distributed (i.i.d.) random variables with zero mean and average energy 
E a = E{|a n | 2 }. Sampling the output signal of the receive filter at times nT yields the 

ISI-free sequence of sample values y n =a n +w n . The noise samples w n are 

complex-valued i.i.d. random variables with zero mean and variance 

a* = E{|w n | 2 } = N 0 . The signal-to-noise ratio becomes SNR = E a /N 0 . The baseline 

10 system then supports a data rate R of 

1 ^NR 

R = J_log 2 (i + ^l) bit/sec. EQ.(2) 
T G 

In EQ. (2), G denotes the SNR gap from capacity. The value of G provides an 
abstract characterization of the effectiveness of particular types of modulation and 
coding schemes, with G = 1 being the theoretical optimum. Assume real systems 

15 operating at BER (bit-error rate) of 1CT 6 . Then, for uncoded modulation G « 8 (9 dB). 
Systems with currently widely employed modulation and coding are characterized by 
G « 4 to 2 (6 to 3 dB). With capacity approaching coding the SNR gap may 
eventually be reduced 1.5 dB or less. 

Sequences of T-spaced quantities q n can be written in polynomial sequence 

20 notation as q(D) = ^ n <; n D n • The delay operator D may be expressed in the frequency 

domain by D = e" j27cfT (= z" 1 ). Using this notation, the sequence of modulation 
symbols becomes a(D) and the sequence of received sample values can generally be 
written as 

y(D) = h(D)a(D) + w(D), EQ. (3) 

25 where h(D) represents the symbol response. In the case of a zero-ISI system, h(D) = 1 
such that y(D) = a(D) + w(D). 
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In light of the above discussion, attempting to increase the modulation rate by 
reducing the amount of excess bandwidth inherently increases filter complexity as 
filters must approach a brick-wall characteristic to maintain zero ISL Clearly, the 
realization of filters exhibiting such brick-wall characteristics may not be pursued. 
5 Therefore, there exists a need in the art to provide a solution that could enable the use 
of a modulation rate 1/T = W in a real communication system without necessitating 
the prohibiting operational constraint described above that requires zero-ISI 
transmission. 
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BRIEF SUMMARY OF THE INVENTION 

The invention described herein may be referred to as decisively reducing 
excess bandwidth to zero, i.e., using "zero excess bandwidth" (ZEB) modulation with 
a modulation of rate 1/T W = W , in order to achieve higher channel throughput. 
5 Transmit and receive filters within actual communication systems are realized 

with smooth spectral roll offs towards zero at the limits of the permitted transmission 
band. These edges now coincide with the edges of the Nyquist band. Thus, the system 
can no longer be a zero-ISI system. There are several techniques known in the art to 
deal with ISI. The method of DFE (decision feedback equalization) has been known 

10 for many years. One problem associated with DFE is error propagation caused by the 
use of incorrect decisions in making subsequent decisions. This problem becomes 
exacerbated with coded modulation. Because with coded modulation reliable 
decisions can only be obtained after sufficient coding delay, for DFE less reliable 
decisions with short or even zero delay must be employed. The approach of TH 

15 (Tomlinson-Harashima) precoding has also been known for over three decades, but TH 
precoding became applied in communication systems only more recently in voiceband 
modems (beginning with V.34 and continuing with later versions) and in SHDSL 
(single-line high-speed digital subscriber links). 

One main advantage of TH precoding over DFE is the lack or error propagation 

20 in the receiver. A second crucial advantage of TH precoding is the possibility to use 
modulation coding in the same way as in zero-ISI systems because of the absence of 
decision feedback filtering in the receiver. Therefore, TH precoding should be 
preferred over DFE whenever possible. 

In general, the scope and spirit of the invention may be described by 

25 intentionally allowing ISI to achieve higher channel throughput and introducing this 
ISI in a form which makes dealing with it in transceivers easy. The use of the known 
technique of TH precoding follows as a necessary consequence. 

The higher channel throughput follows from an analysis of the achievable data 
rate obtained by increasing the modulation rate and accepting lower SNR in return. In 
. 30 an ISI system, SNR is defined at the decision point of a DFE receiver. The lower SNR 
in the ZEB system is a consequence of transmitting less energy per symbol at higher 
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rate and ignoring the power of ISI at the DFE decision point. When the feedback filter 
is transferred to the transmitter for TH precoding, the noise at the decision point of the 
receiver and the spacing between modulation symbols does not change. The 
appropriate SNR definition remains the same as for a DFE receiver. 
5 With reference to EQ. (2) above, the analysis shows that increasing the 

modulation rate outweighs the effect of a lower SNR. The data rate R increases 
linearly with an increase of modulation rate (1/T), but decreases only logarithmically 
with a decrease in SNR. While it that has been known that increasing the modulation 
rate is a principal gating factor for achieving higher data rate across a communication 
10 channel, there has until now been no introduction of a solution that uses ZEB 
modulation because of perceived potentially undesirable effects that may accompany 
it. 

As an example for the rate improvement achievable by ZEB modulation, 
consider first a zero-ISI baseline system employing SQRC filters with a = 0.25. 
15 Assume a signal-to-noise ratio of SNR = 30 dB and coding with an SNR gap to 
capacity of 6 dB (G = 4). From EQ. (2) above, it follows that 7.97 bit per symbol can 
be transmitted at modulation rate 1/T = 0.8 x W such that the bit rate becomes R = 
6.38 bit xW. 

Now consider a ZEB system transmitting signals at modulation rate 1/T W = W 
20 (e.g., where the modulation rate is equal to the available bandwidth). For a fair 
comparison, the ZEB system has to transmit signals with the same PSD as the zero-ISI 
system, i.e., the same spectral shaping by SQRC filters shall be employed. An 
observer measuring the PSD of transmitted signals could not tell a difference between 
the two systems. In the ZEB system, a signal-to-noise ratio of SNR = 27.2 dB is 
25 obtained. According to EQ. (1) above, this permits transmission of 7.04 bit per symbol 
at modulation rate 1/T W = W . Hence, R = 7.04 bit xW. To achieve the same rate - 
improvement in the zero-ISI baseline system through more sophisticated coding, a 
coding scheme with an additional coding gain of 2.64 dB would be needed. Achieving 
such a gain through the ZEB approach instead of resorting to coding and decoding of 
30 much higher complexity can therefore be a very attractive solution. 
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The second aspect of the invention is concerned with a suitable choice of the 
overall channel symbol response h(D) in a ZEB system. This second aspect ensures 
that any introduced ISI is in such a form that it may be dealt with very easily in 
transceivers of a communication system. Because no condition for zero ISI needs to 
5 be satisfied, there is no need to adhere to particular symmetries of the shape of spectral 
roll offs, as is the case with filters implemented using SQRC roll offs. In the literature 
on DFE and TH precoding, h(D) is usually assumed to be a causal monic polynomial, 

h(D) = 1 + h 1 D + h 2 D 2 + — h L D L of some finite degree L 5 corresponding to a finite 
impulse response (FIR). A further aspect of the invention is to let h(D) be a rational 
10 function of D, corresponding to an infinite impulse response (IIR). The simplest such 

( 1 

function with a spectral zero at D = -1 f = ± ~^J anc * a single roll-off parameter p is 

h(D) = = 1 + - = 1 + S D(1 - pD + p 2 D 2 - p 3 D 3 • • • ) EQ. (4) 
1-pD 1-pD 

where 0 < p < 1 and e = 1 - p . The squared magnitude of this IIR response is plotted 
in FIG. 9 for p = 0.95, 0.9, and 0.8 . For s = 2~\ n being an integer, the 

15 implementation of the feedback filter in a TH precoder and a corresponding shaping 
filter as part of the transmit filter are trivial. 

Moreover, within a ZEB modulation communication system, the lack of excess 
bandwidth and the placement of spectral zeroes at the edges of the Nyquist band 
allows almost all, if not all, digital filtering operations to be performed with 1/T- 

20 sampled signals. 
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BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWINGS 

FIG. 1 is a diagram illustrating a prior art communication system supporting a 
modulation rate of (1/T) across a communication channel of bandwidth (W). 

FIG. 2 is a diagram illustrating a prior art approach to support modulation rates 
5 of (1/T) across multiple bands of a communication channel of bandwidth (nW). 

FIG. 3 is a diagram illustrating a prior art, complex baseband model of a zero 
inter-symbol interference (ISI) baseline system. 

FIG. 4A is a diagram illustrating an embodiment of theoretical, un-realizable 
brick wall filtering ensuring spectral nulls at band edges of a spectra portion interest. 
10 FIG. 4B is a diagram illustrating an embodiment of realizable filtering ensuring 

spectral nulls at band edges of a spectra portion interest according to the invention. 

FIG. 5 is a diagram illustrating an embodiment of a communication system 
supporting a ZEB (zero excess bandwidth) modulation rate (1/T W ) across a 
communication channel of bandwidth (W) according to the invention. 
15 FIG. 6 is a diagram illustrating an approach to support ZEB modulation rates 

(1/T W ) across multiple bands of a communication channel of bandwidth (nW) 
according to the invention. 

FIG. 7 is a diagram illustrating an embodiment of a complex baseband model 
of a ZEB communication system with IIR (infinite impulse response) symbol response 
20 that employs TH (Tomlinson-Harashima) precoding according to the invention. 

FIG. 8 is a diagram illustrating an embodiment of squared magnitude of IER 
response for various IIR filter response factors, p , according to the invention. 

FIG. 9 is a diagram illustrating an embodiment of a complex baseband model 
of a ZEB communication system according to the invention. 
25 FIG. 10 is a diagram illustrating an embodiment of a ZEB modulation 

communication transmitter including various forms of encoding and symbol mapping 
and TH precoding according to the invention. 

FIG. 11 is a diagram illustrating an embodiment of a ZEB modulation 
communication receiver according to the invention. 
30 FIG. 12 is a flowchart illustrating an embodiment of a ZEB modulation 

transmit method that is performed according to the invention. 
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FIG. 13 is a flowchart illustrating an embodiment of a ZEB modulation 
method that is performed according to the invention. 
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DETAILED DESCRIPTION OF THE INVENTION 

The invention aims at extending the modulation rate to the actual bandwidth up 
the very band edges of the available bandwidth of a communication channel. A 
motivation for this approach is based on the recognition that modulation rate is the 
5 principal gating factor for achieving the highest data rate across a communication 
channel. To illustrate the leverage of employing a higher modulation rate, a zero ISI 
(inter-symbol interference) QAM (Quadrature Amplitude Modulation) baseline 
communication system having a spectral roll off factor of a = 0.25 and transmitting 8 
bit per symbol may be considered. In this consideration, the transmit power and the 

10 noise power spectral density is assumed to be fixed. 

If the roll off factor could be reduced to a = 0.1, and if the modulation rate is 
increased accordingly, a new zero-ISI communication system may then transmit 7.82 
bit per symbol at the increased modulation rate; this is equivalent to transmitting 8.88 
bit per symbol time when compared to the original zero ISI baseline communication 

15 system with a = 0.25. To achieve a comparable improvement, the original, zero ISI 
baseline communication system would require an additional coding gain of 2.64 dB. 

However, the approach pursued in the present invention is not to achieve rate 
improvements simply through reduction of the spectral roll offs in a zero-ISI system, 
which in the limit for a = 0 would requires unrealizable brick-wall filtering. Instead, 

20 while retaining a given spectral shape of the transmitted signal with a given finite roll 
factor, the modulation rate is increased such that the modulation rate equals the 
available channel bandwidth. The transmitted signal then exhibits spectral zeroes at 
the edges of the Nyquist band, which coincide with the edges of the available 
bandwidth. Hence, such a system exhibits zero excess band bandwidth (ZEB). The 

25 spectral zeroes at the edges of the Nyquist. band cause necessarily ISI, which in the 
proposed ZEB system is dealt with by the known technique of TH (Tomlinson- 
Harashima) precoding. 

The advantage of a ZEB system with TH precoding over a zero-ISI baseline 
system, both systems exhibiting identical transmit spectra with finite roll-off factors, 

30 will become apparent in the following description. 
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FIG. 4A is a diagram illustrating an embodiment of theoretical, un-realizable 
brick wall filtering ensuring spectral nulls at band edges of a spectra portion of 
interest. This theoretical, un-realizable brick wall filtering is shown as having a 
completely flat response throughout the entirety of the communication channel's 
5 bandwidth (W). By using such brick wall filtering, a modulation rate having a 
frequency of 1/T W =W = (1/TX1 + oc) may be employed without any degradation of 
the signal, where T is the modulation rate and a is the spectral roll off factor 
referenced to the prior art approach that employs SQRC (square root raised-cosine) 
filters as described above. This filtering enables all of information within the signal 

10 whose modulation rate is 1/T W = W to be passed through such a brick wall filter, in 
that, the brick wall filter includes the entirety of the spectra of the modulation rate, 
1/T W = W . While a brick wall filtering approach would be ideal to utilize all of the 
available spectra of a communication channel's bandwidth (W), such brick wall filters 
are physically un-realizable. 

15 Therefore, the approach provided by the invention does not attempt to employ 

the theoretical, un-realizable brick wall filtering approach that would be required in a 
zero ISI baseline communication system; that is to say, in a zero ISI baseline 
communication system whose modulation rate is equal to the bandwidth of the 
available band, namely, extended to the very band edges of that available band. 

20 Instead, the invention provides a partial response system with spectral nulls 
(sometimes referred to as spectral zeroes) at the band edges of the available band and 
TH (Tomlinson-Harashima) precoding in the transmitter are included to provide for an 
overall gain in the communication system's throughput. The implementation of a 
realizable filtering approach inherently includes some smoother roll off towards the 

25 band edge frequencies of the available band. 

FIG. 4B is a diagram illustrating an embodiment of realizable filtering ensuring 
spectral nulls at band edges of a spectra portion of interest according to the invention. 
This realizable filtering includes a relatively steep response towards the band edges of 
the communication channel's bandwidth (W). This realizable filtering is shown as 

30 having a relatively flat response throughout the majority of the communication 
channel's bandwidth (W). By using this filtering, a modulation rate having a 
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frequency of 1/T W = W = (l/TXl + cc) may still be employed; again, where T is the 
modulation rate and a is the spectral roll off factor referenced to the prior art approach 
that employs SQRC filters as described above. However, because the filtering 
characteristics do not perfectly span the entirety of the communication channel's 
5 bandwidth (W). As such, a signal whose modulation rate is equal to the available 
bandwidth (W) of the communication channel, after passing through the filtering of the 
FIG. 4B, will incur some errors given the fact that the filter covers less frequency 
spectra than the modulation rate (which spans the entirety of the communication 
channel's bandwidth (W)). 

10 These errors may be quantified as signal loss due to the spectral shaping 

performed by the filtering. In an effort to combat this degradation of signal, a certain 
amount of TH precoding may be performed to the signal. This TH precoding provides 
a coding gain that exceeds the loss of performance experienced from the fact that the 
filtering is of a slightly narrower frequency than the modulation rate. Given that the 

15 coding gain exceeds the signal loss, the overall effect is that a net gain in overall 
system performance may be realized. Various embodiments are described in more 
detail below showing how TH precoding may be implemented, in conjunction to 
modulating a signal at a modulation rate that occupies an entirety of spectra of an 
available band, to provide for an overall improvement and increase in the throughput 

20 and performance within a communication system. 

FIG. 5 is a diagram illustrating an embodiment of a communication system 
supporting a ZEB (zero excess bandwidth) modulation rate (1/T W ) across a 
communication channel of bandwidth (W) according to the invention. Beginning with 
a general depiction of a communication system including a transmitter 

25 communicatively coupled to a receiver via a communication channel, the invention 
operates by transmitting a signal from the transmitter to the receiver, across the 
communication channel, whose modulation rate occupies an entirety of spectra of an 
available band. The available band in this embodiment is the entirety of the 
communication channel's bandwidth (W). The modulation rate has a frequency of 

30 1/T W = W = (1/TX1 + a), where T is the modulation rate and a is the spectral roll off 
factor referenced to the prior art approach that employs SQRC filters as described 
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above. That is to say, the modulation rate of a frequency of 1/T W = W may be 
viewed in the in contrast to the modulation rate employed by a prior art, zero inter- 
symbol interference (ISI) baseline system. 

The increase of the modulation rate according to the invention, by a factor of 
5 (1 + a) when compared to the modulation rate of a prior art, zero ISI baseline system, 
provides for an overall increase on the data rate that may be supported between the 
transmitter and the receiver via the communication channel. However, given the fact 
that a realizable (non-ideal, non-brick wall filter) filter must be employed, there will 
inherently be some errors introduced into the signal. In an effort to counter this 

10 introduction of errors due to the increased modulation rate in conjunction with the 
spectral shaping of a realizable filter that does not exhibit a brick wall filter, TH 
precoding performs, effectively, a channel inversion operation, of the signal before 
modulating it to the modulation rate that occupies an entirety of spectra of an available 
band (which is the communication channel's bandwidth (W) in this embodiment). 

15 Clearly, this TH precoding comes with some cost. TH precoding generally has an 
effect to decrease the overall data rate. However, the reduction in overall data rate 
introduced by the TH precoding is less than the improvement provided by the increase 
of the modulation rate to the entirety of spectra of an available band. The TH 
precoding is operable to compensate for the potential errors that may be introduced 

20 when the modulation rate is increased to the entirety of the spectra of an available 
band. 

FIG. 6 is a diagram illustrating an approach to support ZEB modulation rates 
(1/T W ) across multiple bands of a communication channel of bandwidth (nW) 
according to the invention. This diagram illustrates how there is no spectral spacing 

25 between the sub-bands of the entirety of the communication channel's bandwidth 
(nW), with the exception of the spectral nulls between the sub-bands at the edges of 
each sub-band. This communication channel has a bandwidth (nW) that is partitioned 
into a number of sub-bands. This diagram shows the communication channel of 
bandwidth (nW) being partitioned into n different sub-bands each having bandwidth 

30 (W). To ensure spectral nulls at the edges of the sub-bands, a realizable filter having a 
somewhat smoother (non-brick wall) response is employed. The modulation rate of 
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each sub-band is shown as having a frequency of 1/T W = W = (l/TXl + a), where T is 
the modulation rate and a is the spectral roll off factor again referenced to the prior art 
approach that employs SQRC filters as described above. It is noted here that the 
precoding and/or encoding is operable to compensate for an inherently lower SNR and 
5 potentially undesirably generated ISI between one or more adjacent bands within the 
overall communication channel's bandwidth (W). 

FIG. 7 is a diagram illustrating an embodiment of a complex baseband model 
of a ZEB communication system with an IIR (infinite impulse response) symbol 
response that employs TH (Tomlinson-Harashima) precoding according to the 
10 invention. 

In a transmitter end, information bits that have been encoded and appropriately 
modulated are employed to generate a sequence of discrete-valued modulation 
symbols. It is also noted that some of these information bits may not be encoded, and 
they may be employed as uncoded bits when forming one or more of the symbols of 

15 the sequence of discrete-valued modulation symbols. These symbols of the sequence 
of discrete-valued modulation symbols may be of one or more of a number of types of 
modulation symbols including PAM (pulse amplitude modulation), QPSK (quadrature 
phase shift keying), 16 QAM (quadrature amplitude modulation), 32 QAM (double 
square) (e.g., 2 16 QAM constellations offset from one another in a "double square" 

20 arrangement), 64 QAM, and so on. 

In some instances, combination TCM (Trellis Coded Modulation) coding and 
modulation encoding may be employed to generate a sequence of discrete-valued 
modulation symbols that includes a sequence of trellis coded QAM modulation 
symbols. This sequence of discrete-valued modulation symbols, shown as a(D), is 

25 provided to a TH precoder that employs a simple single pole feedback filter and that 
generates a TH precoded signal, b(D). The operation of the TH precoder may be 
viewed as being an inverse filtering operation of the overall channel response, h p (D), 

of the communication channel. For example, after information bits of an input signal 
have been encoded and appropriately modulated to form the sequence of discrete- 
30 valued modulation symbols, a(D), the TH precoder performs a communication channel 
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inverse filtering operation, 1/h p (D), on the sequence of discrete- valued modulation 

symbols, a(D), thereby generating the TH precoded signal, b(D), which may also be 
viewed as being discrete-time transmit signals. This communication channel inverse 
filtering operation puts the filtered signal outside of a particular signal region (e.g., 
5 beyond the region of the communication channel). Thereafter, the signal is reduced to 
a value such that it fits within the region of interest using a modulo reduction 
operation. In general, the operation of the TH precoder may be viewed as being an 
inverse filtering operation with modulo reduction of the filtered signal. 

Again, the precoded signal, b(D), output from the TH precoder may also be 

10 viewed as being discrete-time transmit signals. This precoded signal, b(D) (e.g., the 
discrete-time transmit signals), is then provided to a DAC (digital to analog converter) 
that converts the precoded signal, b(D), to an continuous time transmit signal whose 
modulation rate is equal to the bandwidth of the available band (e.g., occupies an 
entirety of spectra of the available band). This DAC is clocked at a frequency of nT w , 

15 where n is an integer and T w is the modulation rate that occupies the entire bandwidth 
of the available band. The DAC then generates an analog signal that is then provided 
to a transmit filter. It is also noted that a digital filter may also be interposed between 
the TH precoder and the DAC to allow the operation of the transmit filter (situated 
after the DAC) to operate easier/better in shaping the continuous-time transmit signal 

20 to be launched into the communication channel. 

As with some of the other embodiments of the invention, the entire bandwidth 
of the available band is the same as the entirety of spectra of the communication 
channel into which a signal is launched from the transmitter. This continuous-time 
transmit signal (which is output from the DAC) is then provided to a transmit filter 

25 that performs spectral shaping of the continuous-time transmit signal to ensure spectral 
nulls substantially at band edges of the available band. The combination of the DAC 
and the transmit filter may cooperatively be viewed as being a means to generate a 
continuous-time transmit signal from the discrete-time transmit signals (e.g., from the 
precoded signal, b(D), sometimes referred to as a discrete-time transmit signals). 

30 The filtered, continuous-time transmit signal is launched into the 

communication channel that communicatively couples the transmitter and the receiver 
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such that the filtered, continuous-time transmit signal is transmitted across the 
communication channel from the transmitter to the receiver. The frequency response 
of the communication channel, having a bandwidth (W), is shown as being G c (f) = 1 . 

In addition, as also shown within some of the other embodiments, a certain amount of 
5 noise may also be introduced into the continuous-time transmit signal as it is 
transmitted across the communication channel. In some instances, the noise may be 
represented as having a PSD (power spectral density): N(f) = N 0 (where an AWGN 

(additive white Gaussian noise) communication channel is assumed). 

A receiver at the other end of the communication channel receives the 

10 continuous-time transmit signal that has been transmitted from the transmitter. A 
receive filter within the receiver is implemented to assist in the spectral shaping of the 
received signal to continue to ensure spectral nulls substantially at band edges of the 
available band. The filtering operations performed within the receiver ensure the 
suppression of any noise and undesired signals that are outside of the available band 

15 (e.g., the transmission band employed at the transmitter). These filtering operations 
performed within the receiver also ensure equalization of the signal within the 
transmission band. 

In certain embodiments, the spectral shaping may be performed only in the 
transmit filter of the transmitter. Alternatively, the spectral shaping may be performed 

20 in both the transmit filter of the transmitter and the receive filter of the receiver. This 
received, filtered continuous-time transmit signal is then passed to an ADC (analog to 
digital converter), that is also clocked at the frequency of nT w , where n is an integer 
and T w is the modulation rate that occupies the entire bandwidth of the available 
band), that samples the received (and appropriately filtered) continuous-time transmit 

25 signal thereby generating discrete-time receive signals. These generated discrete-time 
receive signals may undergo further equalization at the modulation rate, if required, to 
ensure that the generated discrete-time receive signals possess the appropriate 
characteristics for estimation of the transmitted sequence of information bearing 
modulation symbols and the information bits encoded therein at the transmitter end of 
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the communication channel. The signal that is output from the ADC is therefore a 
received sample sequence which is described in more detail below. 

Again, as described above in many of the other various embodiments of the 
invention, the modulation rate of a signal transmitted from a transmitter to a receiver 
5 via a communication channel is extended to 1/T W = W , or to the entire spectrum of 
the available band (e.g., equal to the bandwidth of the available band). In this 
embodiment, as in several others, the entirety of spectra of an available band is the 
entirety of spectra of the communication channel itself. Alternatively, the entirety of 
spectra of an available band could be the entirety of spectra of a sub-band of the 

1 0 communication channel . 

For comparison with prior art approaches, in order to operate a prior art zero 
ISI baseline system at this modulation rate (e.g., at modulation rate 1/T W = W), it 
would require the unrealizable brick wall filters (as described above with respect to 
FIG. 4A) in both the transmitter and receiver. As such, the prior art presents no 

15 workable solution by which this may be performed. Clearly, realizable filters should 
inherently exhibit smoother spectral roll offs towards zero at the band edge frequencies 
± W 12 (as described above with respect to FIG. 4B). 

The operation of the invention may be achieved by employing partial response 
signal shaping in combination with (1) decision feedback equalization using a DFE 

20 (decision feedback equalizer) in a receiver or (2) TH (Tomlinson-Harashima) 
precoding in a transmitter. For fixed channels, TH precoding is preferred because it 
avoids the error propagation problem associated when implementing a DFE in a 
receiver, and (more importantly) it is more compatible with modulation coding. There 
are a variety of types of encoding that may be implemented according to the invention 

25 as is described below in other of the various embodiments. 

One may consider a partial response class-I response which exhibits spectral 
nulls (or zeroes) at the band edges of a frequency interval of interest as follows: 

h PR ^(D) = 1 + D,D = e- j27tfTw . 

However, to achieve a data rate improvement over the baseline zero ISI 
30 baseline system, a "fuller" response is needed. 

The symbol response considered in this embodiment is shown below: 
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^( D )=^= 1+ T^= UDh 1( D ) 

p 1+pD 1 + pD 
h p (D) = 1 + D e(l - pD + p 2 D 2 - p 3 D 3 + ...), 

where 0 < p < 1 and 8 = 1 - p . 

This symbol response satisfies the requirements for spectral nulls at the band 
edges of the spectra of interest. This symbol response also provides a flatter spectrum 
5 in most of the channel bandwidth. In addition, this symbol response also lends itself to 
a relative simplicity of implementation. 

FIG. 8 is a diagram illustrating an embodiment of squared magnitude of IIR 
(infinite impulse response) response for various IIR filter response factors, shown as 
p , according to the invention. This diagram illustrates the squared magnitude of the 

10 partial response, h p (D = e~ j27tfN ), versus normalized frequency, f N =^- = fT, (where 

W is the bandwidth of the available band, and T its inverse) for several values of filter 
response factor, p. The following relationship is also noted: 

2 



1+ ZK.i| 2 = 1+ ^ 2 ( 1+ p 2+ p 4+ p 6+ --0=i+^ 



p i+p 

Referring back to FIG. 8, while the feedback filter shown in the TH precoder 
15 illustrated and described above with respect to FIG. 8 is shown as being a simple 
single pole filter, the FIR (finite impulse response) filter that is typically shown in the 
literature in accordance with TH precoding may also be employed without departing 
from the scope and spirit of the invention as well. 

It is noted that the TH precoding does in fact incur some precoding loss. 

20 Precoding loss may be defined as the ratio E b /E a , where E b = E{jb n | 2 } is the average 

energy of the transmitted precoded signals, and E a = E{|a n | 2 } is the average energy of 

the digital signals before undergoing precoding. 

For a quantifiable example of the precoding loss that may be incurred using TH 
precoding, a modulation employing an MxM QAM (Quadrature Amplitude 
25 Modulation) constellation is considered. For M x M -QAM constellations, this 
precoding loss ratio becomes M 2 /(M 2 -1). More specifically, for 64 QAM (M = 6) 
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the ratio is 1.016 (0.068 dB), and for 256 QAM (M = 8) the ratio is 1.0039 (0.017 dB). 
Therefore, the overall precoding losses incurred by performing TH precoding, for 
constellations of this size, are practically negligible. 

It is noted here that the overall improvement in data rate, when employing the 
5 invention, may be analyzed in two different steps. In the first case, the symbol 
response h p (D) is formed entirely in the transmit filter. In this first case, the receive 

filter is still an unrealizable brick wall filter. In the second case, the symbol response 
shaping (e.g., filtering) may be equally split between the transmit filter and the receive 
filter at both ends of the communication channel. 
10 The first case involves spectral shaping that is performed only in the 

transmitter. 

The transmit filter, H T (f), and the receive filter, G R (f), may be characterized as 
follows: 

H t (f) ^j J 2(Ua) h ^ w = e " i27tfTW )' I f l " W /2 ; 0 otherwise 
15 G , R (f) = |^ ^ 1 ^ a) , |f| < W/2; 0 otherwise. 

The filter functions are normalized such that J_ w/2 P T (f)| df = 1/(1 + a) and 

H T(f)GR(f) = T w x h p (D = e~ j2irfTw ). This ensures that the transmit power P T is the 
same as in the zero ISI baseline system described above: 

P - E a = E a 

T (1 + a)T w T ' 
20 The received sample sequence then becomes 

y(D) = h jir (D)b(D) + w'(D) = a(D) + k(D)/M + w'(D) , 

where M is the modulus used by the precoder, and k(D) is a complex integer 
sequence chosen by the precoder such that the real and imaginary parts of the precoded 
signals b n are contained in the interval [-M /2,+ M 12). For MxM QAM 
25 constellations with minimal symbol spacing d 0 , M = d 0 M . 
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The noise samples are uncorrelated and exhibit variance may be described as 
follows: 

2 2(1 + g) 
° w N ° 1 + p • 

From one perspective, the receive filter may be viewed as the optimum 
whitening matched filter for a capacity achieving system, when H^f) is given. The 
signal-to-noise ratio (SNR') relevant for decoding is given as follows: 

SNR' = -^r = SNR- 1 + P 



< • 2(1 + a) 5 
where SNR is the signal-to-noise ratio of the baseline system. The rate 
achievable with an SNR-gap of G then becomes 

10 R' = — log/ 1 + ^^] bit/sec. 

T w v G J 

For comparison with the baseline system, the modulation rate must be 
expressed in bits per symbol time T of the baseline system, as follows 



R'T = (1 + a)log, 



f A SNR 1 + p 
1+ 1 



bit, 



G 2(l + a)J 

Therefore, it can be seen that the overall performance of throughput of the 
15 communication system is improved by increasing the modulation rate. 

The second case involves spectral shaping in both the transmitter and the 
receiver. 

The transmit filter, H T (f), and the receive filter, G R (f), may be now 
characterized as follows: 



H T(0 = Jj7T^-VKD = e H2 ^), |f|< W/2; 0 otherwise, 
[V(1 + a)g 

Gp(f) = {v/T w (1 + a)g Vh(D = e" j27lfTw ), |f| < W/2 ; 0 otherwise. 
To achieve the same transmit power as in the zero IS I baseline system 
described above, g is determined such that J^^jH^f )| 2 df = 1/(1 + a) : 
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g = T w J | h(D = e j2KfTw )| df (by numerical integration). 



The variance of the noise samples to the receive-filter output is given by 



a; 2 =N 0 (1 + a)gT w J | h jir (D = e- j2wfT )| df = N 0 (1 + a)g 2 , 



-W/2 



so that 



5 



SNR"=% = 




SNR 



9 2 (1 + a)' 



Because of the spectral nulls (or zeroes) of the receive filter at the band edges, 
the noise is not entirely uncorrelated. However, the non-whiteness is limited to a 
narrow frequency range. Ignoring this fact, the rate per symbol time T of the baseline 
system becomes 



Again, it can be seen that the overall performance of throughput of the 
communication system is improved by increasing the modulation rate. 

While the zero ISI baseline system described above has been used in 
comparison to illustrate the improvements provided by the invention, it is also noted 

15 that a ZEB (zero excess bandwidth) modulation communication system may 
nevertheless be realized using SQRC filters with the exception that the phase 
characteristic of those SQRC filters is modified such that the overall symbol response 
becomes causal. Alternatively, the required phase change could be achieved by an 
adaptive equalizer. By doing this, the same SQRC transmit and SQRC phase filters 

20 could be used as in an existing zero ISI baseline communication system. The main 
difference is that the zero ISI baseline communication system operates at a modulation 
rate of 1/T = W /(1 + a), whereas the ZEB modulation communication system operates 
at a modulation rate of1/T w = W = (1 / TXl + a) . 



10 



R"T = (1 + a)log 2 1 + 



SNR 1 

G g 2 (l + (x) 



7 



bit. 



This analysis requires the spectral factorization shown as follows: 



25 
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where H rc (f,oc) is a RC (raised cosine) spectrum with spectral nulls at 
f = ±W 12 and roll off factor, a . The polynomial h rc (D) has to be monic and causal, 
and represents the overall system response to be achieved jointly by the transmit filter 
and the receive filter. The amplitude characteristics of H rc (f,ot) and h rc (D = e" j2Ttffyv ) 
5 differ only by a scaling factor. 

The factorization approach for determining h rc (D) and A 2 can be found in 

Appendix III of G. D. Forney and G. Ungerboeck, "Modulation and coding for linear 
Gaussian channels" IEEE Trans, Inform. Theory, vol. 44, pp. 2384-2415, October 
1998. 

10 The transmit filter and the receive filter must be such that 

Hj(f)G R (f)=T w h rc (e ~ i2rcfTw ). If the spectral shaping is equally split between the 

transmit filter and the receive filter, then both filters have the characteristic of a SQRC 
filter. It is therefore noted that a ZEB modulation communication system, employing 
SQRC filters within the transmit filter and the receive filter, has a marked performance 

15 improvement over a zero ISI baseline communication system that also employs SQRC 
filters within the transmit filter and the receive filter. The increased modulation rate of 
the ZEB modulation communication system, coupled with TH precoding, provides for 
an increase in the data throughput of the communication system. 

FIG. 9 is a diagram illustrating an embodiment of a complex baseband model 

20 of a ZEB communication system according to the invention. Again, as described in 
some of the other embodiments, a signal generated within a transmitter is launched 
across a communication channel to a receiver. The communication channel will 
typically introduce a certain degree of noise having a certain PSD (power spectral 
density). 

25 Within the transmitter, a signal that includes a sequence of discrete-valued 

modulation symbols is provided to a DAC (digital to analog converter) that converts 
the sequence of modulation symbols into discrete-time transmit signals. The DAC 
operates by inserting the discrete-time transmit signals at the modulation rate (which is 
equal to the bandwidth of the available band) thereby generating a continuous-time 

30 transmit signal. Within this embodiment as within others described herein, the entire 

23 



DOCKET NO. BP2431 

bandwidth of the available band is the entirety of spectra of the communication 
channel itself. This continuous-time transmit signal is then provided to a transmit 
filter that performs spectral shaping of thereof to ensure spectral nulls (or zeroes) 
substantially at band edges of the available band; the output of the transmit filter may 
5 be viewed as being a filtered, continuous-time transmit signal in this embodiment. The 
filtered, continuous-time transmit signal is then launched into the communication 
channel that communicatively couples the transmitter to the receiver, and the filtered, 
continuous-time transmit signal is thereby transmitted from the transmitter to the 
receiver via the communication channel. 

10 Within the receiver, -a receive filter may be implemented to assist in the spectral 

shaping of the received continuous-time signal (e.g., a continuous-time receive signal) 
to continue to ensure spectral nulls (or zeroes) exist substantially at the band edges of 
the available band. In certain embodiments, the spectral shaping may be performed 
only in the transmitter. Alternatively, the spectral shaping may be performed in both 

15 the transmit filter of the transmitter and the receive filter of the receiver. This filtered, 
continuous-time receive signal is then passed to an ADC (analog to digital converter) 
that samples the filtered, continuous-time receive signal at the modulation rate that is 
equal to the bandwidth of the available band to generate discrete-time receive signals. 
In this embodiment, in an effort to correct for any possible errors that may have 

20 been introduced into the signal during transmission across the communication channel, 
a DFE (decision feedback equalizer) performs decision feedback equalization on the 
discrete-time receive signals. This DFE operates to ensure that the discrete-time 
receive signals possess the appropriate characteristics such that these discrete-time 
receive signals may subsequently undergo appropriate decoding to estimate the 

25 transmitted sequence of information bearing modulation symbols and outputting the 
estimated information bits encoded therein. 

There are a variety of ways in which a DFE may be implemented. However, at 
a very minimum a DFE structure includes a feed forward filter that receives input, a 
decision block that makes hard decisions of the input that has been feed forward 

30 filtered, and a feedback filter that adds an error signal (corresponding to previous hard 
decisions that have been appropriately weighted/filtered by the feedback filter) into the 
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output from the feed forward filter. In doing this, any ISI that may have been 
undesirably introduced into the symbols that are provided to the DFE may be canceled 
out at the output of the feed forward filter by subtracting past symbol values that have 
been appropriately weighted. 
5 In other words, previous hard decisions are fed back to influence subsequent 

hard decisions within the DFE in an effort to combat and (ideally) eliminate 
undesirable effects such as ISI. Cooperatively, the operation of the feed forward filter, 
the decision block, and the feedback filter are typically described as being a DFE 
structure. It is noted here that the operation of the invention may be achieved by 

10 employing partial response signal shaping in combination with either (1) decision 
feedback equalization using a DFE in the receive end of the communication channel or 
(2) TH precoding in the transmit end of the communication channel. The embodiment 
shown in the FIG. 9, using a DFE structure, may be implemented as an alternative 
embodiment of the invention when TH precoding is not as desirable in the transmit end 

15 of a communication system. 

FIG. 10 is a diagram illustrating an embodiment of a ZEB modulation 
communication transmitter including various forms of encoding and symbol mapping 
and TH precoding according to the invention. The communication transmitter shown 
in this embodiment supports any one of any number of types of encoding of 

20 information bits. In addition, whichever encoding is implemented in this embodiment 
is performed in combination with TH (Tomlinson-Harashima) precoding in the 
transmitter as well. 

Information bits are provided to the transmitter that is operable perform 
appropriate encoding and TH precoding before launching an appropriately filtered, 
25 continuous-time transmit signal into a communication channel. After the signal has 
been launched into the communication channel from the transmitter, the signal is 
communicated from the transmitter to a receiver at the other end of the communication 
channel. The communication channel may introduce a certain degree of noise having a 
particular PSD. 

30 Within the transmitter, encoding of these information bits is performed using an 

encoder and symbol mapper. This encoder and symbol mapper may be implemented 
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to support uncoded modulation, TCM (trellis coded modulation), TTCM (turbo trellis 
coded modulation), LDPC (low density parity check) encoding and modulation, as 
well as any other types of encodings that operate to counter the effects of lowered SNR 
and potentially introduced ISI that may be generated by the increase of the modulation 
5 rate of the signal to a value that is equal to the bandwidth of the available band. 

After the information bits have been appropriately encoded using whichever 
encoding means is employed within a particular embodiment, the encoded bits may be 
grouped to form symbols that may be symbol mapped according to any number of 
different types of modulations (where each modulation includes a constellation shape 

10 and unique mapping of the constellation points included therein). For examples, in 
one embodiment, rate 3/4 turbo encoding may be employed in accordance with TTCM 
by the encoder and symbol mapper such that 4 encoded bits are output for every 3 
information bits that are input to the encoder. These 4 encoded bits may be grouped to 
form a symbol that is symbol mapped according to a modulation that employs 4 bit 

15 symbols (e.g., a 16 QAM modulation that includes a 16 QAM shaped constellation, 
having 16 constellation points, and has a unique mapping for each of the constellation 
points included therein). Also, other encoding techniques may also be implemented 
without departing from the scope and spirit of the invention including puncturing of 1 
or more of the encoded bits that are output from the encoder, rearranging the order of 1 

20 or more of the encoded bits that are output from the encoder, and so on. 

After the information bits have been appropriately encoded and symbol mapped 
thereby generating a sequence of discrete-valued modulation symbols, then this 
resulting sequence of discrete-valued modulation symbols is provided to a TH 
(Tomlinson-Harashima) precoder that operates in the manner as described above 

25 within other of the various embodiments. Again, the operation of the TH precoder 
may be viewed as being an inverse filtering operation of the overall channel response 
of the communication channel. 

When a ZEB modulation communication transmitter employs any one of the 
types of encodings described herein, a corresponding ZEB modulation communication 

30 receiver (that receives a signal transmitted from the ZEB modulation communication 
transmitter via a communication channel) is then operable to perform the appropriately 
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corresponding decoding of such an encoded and transmitted signal. In addition, the 
receiver knows which modulation type (including the constellation shape and mapping 
of the constellation points therein) has been employed by encoder and symbol mapper 
of the transmitter, so that the receiver may appropriately decode the transmitted 
5 sequence of information bearing symbols sent from the transmitter. 

For example, when the ZEB modulation communication transmitter employs 
uncoded modulation in conjunction with a particular symbol mapping of a 
constellation, then a ZEB modulation communication receiver is operable to decoding 
of such a sequence of discrete-valued uncoded modulation symbols with knowledge of 

10 the manner in which the information bits have been symbol mapped in the encoder and 
symbol mapper within the transmitter. 

Similarly, when the ZEB modulation communication transmitter employs TCM 
within the encoder and symbol mapper, then a ZEB modulation communication 
receiver is operable to decoding of such a sequence of discrete-valued uncoded 

15 modulation symbols with knowledge of the manner in which the information bits have 
been encoded and symbol mapped in the encoder and symbol mapper within the 
transmitter according to the particular implementation of TCM. 

In short, a corresponding ZEB modulation communication receiver is operable 
to perform appropriate decoding of a received signal with knowledge of the particular 

20 manner in which information bits had been encoded and symbol mapped in the 
encoder and symbol mapper within the transmitter. 

It is also noted that the encoding and symbol mapping within the transmitter 
may be performed using any of a variety of modulation types including PAM (pulse 
amplitude modulation), QPSK (quadrature phase shift keying), 16 QAM (quadrature 

25 amplitude modulation), 32 QAM (double square) (e.g., 2 16 QAM constellations offset 
from one another in a "double square" arrangement), 64 QAM, and so on. 

Within the transmitter shown in this embodiment, the sequence of discrete- 
valued modulation symbols is input to the TH precoder where that sequence is 
conditioned to generate discrete-time transmit signals. These discrete-time transmit 

30 signals are then provided to a DAC that operates by inserting the discrete-time 
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transmit signals at a modulation rate that is equal to the bandwidth of the available 
transmission band thereby generating a continuous-time transmit signal. 

In one embodiment, the bandwidth of the available transmission band is the 
entire bandwidth of the communication channel itself. In another embodiment, the 
5 bandwidth of the available transmission band is the entire bandwidth of a sub-band of 
the communication channel. The resulting continuous-time transmit signal is then 
provided to a transmit filter that performs spectral shaping of the continuous-time 
transmit signal to ensure that spectral nulls (or zeroes) are located substantially at the 
band edges of the allowed transmission band. Ultimately, the resulting, filtered 

10 continuous-time transmit signal is then launched into the communication channel. 

FIG. 11 is a diagram illustrating an embodiment of a ZEB modulation 
communication receiver according to the invention. The ZEB modulation 
communication receiver in this embodiment may be viewed as being a ZEB 
modulation communication receiver that operates in conjunction with the 

15 corresponding ZEB modulation communication transmitter described above with 
respect to the FIG. 10. 

A filtered, continuous-time signal that has been transmitted across a 
communication channel (from a ZEB communication transmitter) is received by the 
ZEB modulation communication receiver. At the receiver end of the communication 

20 channel, this received continuous-time signal may be referred to as a continuous-time 
receive signal. The continuous-time receive signal is provided to a receive filter that 
may be implemented to perform spectral shaping of the continuous-time receive signal 
to ensure suppression of noise and any other undesired signals outside of the 
transmission band and also to ensure equalization of the continuous-time receive signal 

25 within the transmission band. Moreover, this receive filter may operate to perform 
spectral shaping, in conjunction with a transmit filter at the transmitter end of the 
communication channel (e.g., in conjunction with the transmit filter within the ZEB 
communication transmitter), to ensure that spectral nulls (or zeroes) exist at the band 
edges of the allowed transmission band. As described above within other 

30 embodiments, the spectral shaping that is performed to ensure these spectral nulls at 
the band edges of the available transmission band may be performed in the transmitter 
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alone (using a transmit filter) or in both the transmitter and the receiver (cooperatively 
using both a transmit filter and a receive filter). 

This now filtered, continuous-time receive signal (output from the receive 
filter) is then sampled using an ADC (analog to digital converter) at the modulation 
5 rate thereby obtaining discrete-time receive signals. Again, this modulation rate is 
equal to the bandwidth of the available band. 

These obtained discrete-time receive signals are then provided to a decoder 
where they undergo appropriately corresponding decoding. That is to say, these 
discrete-time receive signals now undergo decoding in a manner that corresponds to 

10 the manner in which information bits have been encoded and symbol mapped at the 
transmitter end of the communication channel (e.g., in the ZEB communication 
transmitter). For example, the decoder may be implemented to perform TCM 
decoding, TTCM decoding, LDPC decoding, or any number of other types of 
decodings that correspond to manner in which information bits are encoded at the 

15 transmitter end of the communication channel. 

If required, additional equalization (usually implemented as adaptive 
equalization) may be performed (after the sampling that is performed by the ADC) at 
that same modulation rate to ensure that the received signal has the characteristics that 
are required for subsequent estimation of the transmitted sequence of information 

20 bearing modulation symbols and the information bits encoded therein. 

FIG. 12 is a flowchart illustrating an embodiment of a ZEB modulation 
transmit method that is performed according to the invention. Initially, the method 
involves receiving information bits. The method then involves encoding those 
information bits into a sequence of discrete-valued modulation symbols. These 

25 encoded bits may be generated using any of the various means of encoding described 
herein (e.g., including TCM, TTCM, LDPC, and so on). This may also involve 
employing uncoded information bits when forming the sequence of discrete-valued 
modulation symbols. This step also involves performing any appropriate symbol 
mapping of the encoded bits (and/or uncoded information bits) when generating this 

30 sequence of discrete- valued modulation symbols. 
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The method then performs TH (Tomlinson-Harashima) precoding of the 
sequence of discrete- valued modulation symbols according to overall channel for ZEB 
modulation with spectral nulls at band edges thereby generating discrete-time transmit 
signals. The method then involves inserting these discrete-time transmit signals at a 
5 modulation rate equal to available bandwidth into means to generate a continuous-time 
transmit signal. This means also performs filtering that creates spectral nulls (or 
zeroes) at the edges of the available transmission band, where these edges coincide 
with the edge of the Nyquist band. This means may be implemented using a DAC 
(digital to analog converter) in conjunction with a transmit filter to generate a filtered, 

10 continuous-time transmit signal. The method then operates by launching the filtered, 
continuous-time transmit signal into a communication channel. This filtered, 
continuous-time transmit signal may then be transmitted from a communication 
transmitter to a communication receiver via the communication channel. 

FIG. 13 is a flowchart illustrating an embodiment of a ZEB modulation receive 

15 method that is performed according to the invention. Initially, the method operates by 
receiving a continuous-time signal from a communication channel (this received signal 
may be referred to as a continuous-time receive signal). 

The method may then operate by performing filtering operations to ensure 
suppression of signals and noise outside of the desired signal band and to contribute to 

20 shaping of the desired signal into an appropriate form for decoding. These filtering 
operations may be implemented to operate cooperatively with spectral shaping that is 
performed within a ZEB modulation transmit method. 

The method then continues by sampling the signal thereby obtaining discrete- 
time transmit signals at the modulation rate. The method may also involve performing 

25 additional (e.g., usually "adaptive") equalization to ensure that the received signal 
exhibits characteristics assumed at the transmitter end of the communication channel 
for TH precoding. The method ultimately involves performing decoding to estimate 
the transmitted sequence of information bearing modulation symbols and outputting 
the corresponding information bits. 

30 It is also noted that the various methods described above and corresponding to 

FIG. 12 and FIG. 13 may be performed within any number of communication devices, 
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including (where appropriate) communication transmitters, communication receivers, 
and/or communication transceivers. 

The invention illustrates the importance and benefits that may be realized when 
choosing the highest possible modulation rate (or symbol rate) possible. In a given 
5 spectra of interest within an available band, this includes the entirety of the spectra of 
that available band. In various instances, this includes the entirety of the spectra of a 
sub-band of a communication channel's bandwidth or the entire communication 
channel's bandwidth itself. 

Some final remarks are provided here to quantify some of the many 
10 performance benefits provided by the invention. A ZEB modulation communication 
system with IIR response h p (D) = (1 + D)/(1 + pD) p = 0.9 , achieves a rate increase of 

1.5 bit compared to a zero ISI baseline communication system operating at 8 
bits/symbol using SQRC filters with spectral roll off factors of a = 0.25 . This is the 
spectral roll off factor used for upstream transmission within the DOCSIS (Data Over 

15 Cable Service Interface Specification) 1.0/1.2 cable systems. To achieve this same 
rate increase in a zero ISI baseline communication system, the SNR would have to be 
increased by at least 4.5 dB. Alternatively, an additional coding gain (SNR- gap 
reduction) of 4.5 dB would be needed to provide for such improved performance 
within a zero ISI baseline communication system. The rate improvements are still 1.0 

20 bit (or 3 dB) over such a zero ISI baseline communication system with a spectral roll 
off factor of a = 0.18 , and the rate improvements are even 0.65 bit (or 1.95 dB) over 
such a zero ISI baseline communication system with a spectral roll off factor of 
a = 0.12. These are the spectral roll off factors employed in the ITU (International 
Telecommunication Union) J.83-Annex B modes for downstream transmission in cable 

25 systems (e.g., a = 0.18 for the 64 QAM operational mode, and a = 0.12 for the 256 
QAM operational mode). The complexity of the digital signal processing portion of 
such a ZEB modulation communication system, according to the invention, is 
relatively modest in view of the performance improvements that it provides. 

When implementing such a ZEB modulation communication system, the 

30 designer must ensure that the transmit filter and the receive filter are designed with 
sufficiently steep roll offs near the band edges of the available band (and ensuring 
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spectral nulls at those band edges as well). Even when a design of such a ZEB 
modulation communication system is implemented using SQRC transmit and receive 
filters (as is sometimes done in many prior art zero ISI baseline communication 
systems), the description herein has shown that a performance improvement may 
5 nevertheless be realized. The difference in such a ZEB modulation communication 
system employing SQRC transmit and receive filters (compared to a zero ISI baseline 
communication systems employing SQRC transmit and receive filters) is that the ZEB 
modulation communication system operates using a modulation rate that occupies an 
entirety of spectra of an available band. In the case where this is a communication 

10 channel's bandwidth (W), the ZEB modulation communication system operates using 
the modulation rate of 1/T W =W, whereas the zero ISI baseline communication 
system operates using the modulation rate of 1/T = W/(1 + a). In this case, a rate 
improvement of 0.8 bit (or 2.4 dB) is found for a spectral roll off factor of a = 0.25 . 
Although such a rate improvement in this embodiment is less than the 1.5 bit (or 4.5 

15 dB) rate increase that is realized when in a ZEB modulation communication system 
with IIR response h p (D)= (1 + D)/(1 + pD), p = 0.9 that does not use SQRC transmit 

and receive filters, the rate increase is still well worth pursuing. ^ 

It is also noted that adaptive equalization is sometimes needed in 
communication systems for high spectral efficiencies (e.g., > 4 bits/symbol), even for 

20 well defined communication channels. Front end filters are sometimes not easily 
realized with sufficient precision. In the event that there is no excess bandwidth and 
the signal spectrum rolls off to zero at the band edges of the available band, then 
adaptive equalization may be achieved by a symbol spaced equalizer for arbitrary 
sampling phases of the received signal. This may be viewed as yet another of the 

25 benefits provided by the invention. 

In view of the above detailed description of the invention and associated 
drawings, other modifications and variations will now become apparent. It should also 
be apparent that such other modifications and variations may be effected without 
departing from the spirit and scope of the invention. 
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